{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Optional Lab - Softmax Function\n",
    "In this lab, we will explore the softmax function. This function is used in both Softmax Regression and in Neural Networks when solving Multiclass Classification problems.  \n",
    "\n",
    "<center>  <img  src=\"./images/C2_W2_Softmax_Header.PNG\" width=\"600\" />  <center/>\n",
    "\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('./deeplearning.mplstyle')\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "from IPython.display import display, Markdown, Latex\n",
    "from sklearn.datasets import make_blobs\n",
    "%matplotlib widget\n",
    "from matplotlib.widgets import Slider\n",
    "from lab_utils_common import dlc\n",
    "from lab_utils_softmax import plt_softmax\n",
    "import logging\n",
    "logging.getLogger(\"tensorflow\").setLevel(logging.ERROR)\n",
    "tf.autograph.set_verbosity(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note**: Normally, in this course, the notebooks use the convention of starting counts with 0 and ending with N-1,  $\\sum_{i=0}^{N-1}$, while lectures start with 1 and end with N,  $\\sum_{i=1}^{N}$. This is because code will typically start iteration with 0 while in lecture, counting 1 to N leads to cleaner, more succinct equations. This notebook has more equations than is typical for a lab and thus  will break with the convention and will count 1 to N."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Softmax Function\n",
    "In both softmax regression and neural networks with Softmax outputs, N outputs are generated and one output is selected as the predicted category. In both cases a vector $\\mathbf{z}$ is generated by a linear function which is applied to a softmax function. The softmax function converts $\\mathbf{z}$  into a probability distribution as described below. After applying softmax, each output will be between 0 and 1 and the outputs will add to 1, so that they can be interpreted as probabilities. The larger inputs  will correspond to larger output probabilities.\n",
    "<center>  <img  src=\"./images/C2_W2_SoftmaxReg_NN.png\" width=\"600\" />  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The softmax function can be written:\n",
    "$$a_j = \\frac{e^{z_j}}{ \\sum_{k=1}^{N}{e^{z_k} }} \\tag{1}$$\n",
    "The output $\\mathbf{a}$ is a vector of length N, so for softmax regression, you could also write:\n",
    "\\begin{align}\n",
    "\\mathbf{a}(x) =\n",
    "\\begin{bmatrix}\n",
    "P(y = 1 | \\mathbf{x}; \\mathbf{w},b) \\\\\n",
    "\\vdots \\\\\n",
    "P(y = N | \\mathbf{x}; \\mathbf{w},b)\n",
    "\\end{bmatrix}\n",
    "=\n",
    "\\frac{1}{ \\sum_{k=1}^{N}{e^{z_k} }}\n",
    "\\begin{bmatrix}\n",
    "e^{z_1} \\\\\n",
    "\\vdots \\\\\n",
    "e^{z_{N}} \\\\\n",
    "\\end{bmatrix} \\tag{2}\n",
    "\\end{align}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which shows the output is a vector of probabilities. The first entry is the probability the input is the first category given the input $\\mathbf{x}$ and parameters $\\mathbf{w}$ and $\\mathbf{b}$.  \n",
    "Let's create a NumPy implementation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_softmax(z):\n",
    "    ez = np.exp(z)              #element-wise exponenial\n",
    "    sm = ez/np.sum(ez)\n",
    "    return(sm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, vary the values of the `z` inputs using the sliders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.close(\"all\")\n",
    "plt_softmax(my_softmax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you are varying the values of the z's above, there are a few things to note:\n",
    "* the exponential in the numerator of the softmax magnifies small differences in the values \n",
    "* the output values sum to one\n",
    "* the softmax spans all of the outputs. A change in `z0` for example will change the values of `a0`-`a3`. Compare this to other activations such as ReLU or Sigmoid which have a single input and single output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Cost\n",
    "<center> <img  src=\"./images/C2_W2_SoftMaxCost.png\" width=\"400\" />    <center/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss function associated with Softmax, the cross-entropy loss, is:\n",
    "\\begin{equation}\n",
    "  L(\\mathbf{a},y)=\\begin{cases}\n",
    "    -log(a_1), & \\text{if $y=1$}.\\\\\n",
    "        &\\vdots\\\\\n",
    "     -log(a_N), & \\text{if $y=N$}\n",
    "  \\end{cases} \\tag{3}\n",
    "\\end{equation}\n",
    "\n",
    "Where y is the target category for this example and $\\mathbf{a}$ is the output of a softmax function. In particular, the values in $\\mathbf{a}$ are probabilities that sum to one.\n",
    ">**Recall:** In this course, Loss is for one example while Cost covers all examples. \n",
    " \n",
    " \n",
    "Note in (3) above, only the line that corresponds to the target contributes to the loss, other lines are zero. To write the cost equation we need an 'indicator function' that will be 1 when the index matches the target and zero otherwise. \n",
    "    $$\\mathbf{1}\\{y == n\\} = =\\begin{cases}\n",
    "    1, & \\text{if $y==n$}.\\\\\n",
    "    0, & \\text{otherwise}.\n",
    "  \\end{cases}$$\n",
    "Now the cost is:\n",
    "\\begin{align}\n",
    "J(\\mathbf{w},b) = - \\left[ \\sum_{i=1}^{m} \\sum_{j=1}^{N}  1\\left\\{y^{(i)} == j\\right\\} \\log \\frac{e^{z^{(i)}_j}}{\\sum_{k=1}^N e^{z^{(i)}_k} }\\right] \\tag{4}\n",
    "\\end{align}\n",
    "\n",
    "Where $m$ is the number of examples, $N$ is the number of outputs. This is the average of all the losses.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensorflow\n",
    "This lab will discuss two ways of implementing the softmax, cross-entropy loss in Tensorflow, the 'obvious' method and the 'preferred' method. The former is the most straightforward while the latter is more numerically stable.\n",
    "\n",
    "Let's start by creating a dataset to train a multiclass classification model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# make  dataset for example\n",
    "centers = [[-5, 2], [-2, -2], [1, 2], [5, -2]]\n",
    "X_train, y_train = make_blobs(n_samples=2000, centers=centers, cluster_std=1.0,random_state=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The *Obvious* organization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model below is implemented with the softmax as an activation in the final Dense layer.\n",
    "The loss function is separately specified in the `compile` directive. \n",
    "\n",
    "The loss function `SparseCategoricalCrossentropy`. The loss described in (3) above. In this model, the softmax takes place in the last layer. The loss function takes in the softmax output which is a vector of probabilities. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "model = Sequential(\n",
    "    [ \n",
    "        Dense(25, activation = 'relu'),\n",
    "        Dense(15, activation = 'relu'),\n",
    "        Dense(4, activation = 'softmax')    # < softmax activation here\n",
    "    ]\n",
    ")\n",
    "model.compile(\n",
    "    loss=tf.keras.losses.SparseCategoricalCrossentropy(),\n",
    "    optimizer=tf.keras.optimizers.Adam(0.001),\n",
    ")\n",
    "\n",
    "model.fit(\n",
    "    X_train,y_train,\n",
    "    epochs=10\n",
    ")\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the softmax is integrated into the output layer, the output is a vector of probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_nonpreferred = model.predict(X_train)\n",
    "print(p_nonpreferred [:2])\n",
    "print(\"largest value\", np.max(p_nonpreferred), \"smallest value\", np.min(p_nonpreferred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preferred <img align=\"Right\" src=\"./images/C2_W2_softmax_accurate.png\"  style=\" width:400px; padding: 10px 20px ; \">\n",
    "Recall from lecture, more stable and accurate results can be obtained if the softmax and loss are combined during training.   This is enabled by the 'preferred' organization shown here.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the preferred organization the final layer has a linear activation. For historical reasons, the outputs in this form are referred to as *logits*. The loss function has an additional argument: `from_logits = True`. This informs the loss function that the softmax operation should be included in the loss calculation. This allows for an optimized implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "preferred_model = Sequential(\n",
    "    [ \n",
    "        Dense(25, activation = 'relu'),\n",
    "        Dense(15, activation = 'relu'),\n",
    "        Dense(4, activation = 'linear')   #<-- Note\n",
    "    ]\n",
    ")\n",
    "preferred_model.compile(\n",
    "    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),  #<-- Note\n",
    "    optimizer=tf.keras.optimizers.Adam(0.001),\n",
    ")\n",
    "\n",
    "preferred_model.fit(\n",
    "    X_train,y_train,\n",
    "    epochs=10\n",
    ")\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Output Handling\n",
    "Notice that in the preferred model, the outputs are not probabilities, but can range from large negative numbers to large positive numbers. The output must be sent through a softmax when performing a prediction that expects a probability. \n",
    "Let's look at the preferred model outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_preferred = preferred_model.predict(X_train)\n",
    "print(f\"two example output vectors:\\n {p_preferred[:2]}\")\n",
    "print(\"largest value\", np.max(p_preferred), \"smallest value\", np.min(p_preferred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output predictions are not probabilities!\n",
    "If the desired output are probabilities, the output should be be processed by a [softmax](https://www.tensorflow.org/api_docs/python/tf/nn/softmax)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm_preferred = tf.nn.softmax(p_preferred).numpy()\n",
    "print(f\"two example output vectors:\\n {sm_preferred[:2]}\")\n",
    "print(\"largest value\", np.max(sm_preferred), \"smallest value\", np.min(sm_preferred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select the most likely category, the softmax is not required. One can find the index of the largest output using [np.argmax()](https://numpy.org/doc/stable/reference/generated/numpy.argmax.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(5):\n",
    "    print( f\"{p_preferred[i]}, category: {np.argmax(p_preferred[i])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SparseCategorialCrossentropy or CategoricalCrossEntropy\n",
    "Tensorflow has two potential formats for target values and the selection of the loss defines which is expected.\n",
    "- SparseCategorialCrossentropy: expects the target to be an integer corresponding to the index. For example, if there are 10 potential target values, y would be between 0 and 9. \n",
    "- CategoricalCrossEntropy: Expects the target value of an example to be one-hot encoded where the value at the target index is 1 while the other N-1 entries are zero. An example with 10 potential target values, where the target is 2 would be [0,0,1,0,0,0,0,0,0,0].\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!\n",
    "In this lab you \n",
    "- Became more familiar with the softmax function and its use in softmax regression and in softmax activations in neural networks. \n",
    "- Learned the preferred model construction in Tensorflow:\n",
    "    - No activation on the final layer (same as linear activation)\n",
    "    - SparseCategoricalCrossentropy loss function\n",
    "    - use from_logits=True\n",
    "- Recognized that unlike ReLU and Sigmoid, the softmax spans multiple outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
